Evaluating Spoken Language Systems

Spoken language systems (SLSs) for accessing information sources or services through the telephone network and the Internet are currently being trialed and deployed for a variety of tasks. Evaluating the usability of different interface designs requires a method for comparing performance of different versions of the SLS. Recently, Walker et al (1997) proposed PARADISE (PARAdigm for DIalogue System Evaluation) as a general methodology for evaluating SLSs. The PARADISE framework models user satisfaction with an SLS as a linear combination of measures reflecting both task success and dialogue costs. As a test of this methodology, we applied PARADISE to dialogues collected with three SLSs. This paper describes the salient measures identified using PARADISE within and across the three SLSs, and discusses the generalizability of PARADISE performance models.